Perspective improvement for image and video applications

ABSTRACT

A system and method for reducing distance-based distortion in a camera image of an object, where the distanced-based distortion is due to differences in distance from the camera to different parts of the object. In one approach, the distortion is reduced by estimating distances to different parts of the object and then generating a reprojected image of the object dependent upon the estimated distances and upon a virtual viewpoint that is more distant than the camera from the object. In a further approach, the image is warped such that points in the image match corresponding points in one or more stored templates. In a still further approach, if excessive distortion is present in the image, the camera zoom is increased and a magnified image is displayed, prompting a person to move farther from the camera thereby reducing the distortion.

RELATED APPLICATIONS

This patent application is related to U.S. application Ser. No.11/380,196, entitled Perspective Improvement for Image and VideoApplications, priority to which is claimed.

BACKGROUND

Many camera based products, such as videophones, webcams and mobiledevices, require wide angle lenses. In order to avoid having their faceslook too small, users tend to move their faces close to the camera. As aresult, perspective distance effects become large and their faces appeardistorted in the image, with large noses and pinched cheeks. Users findthis unflattering. This distance-based perspective effect is anundesirable feature of videophone and video email products.

There are other perspective effects such as wide-angle (fish-eye)distortion and oblique distortion (which causes keystoning).Compensation techniques have been proposed for mitigating these effects,but these techniques are not applicable to the type of distortiondescribed above.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the invention are setforth in the appended claims. The invention itself, however, as well asthe preferred mode of use, and further objects and advantages thereof,will best be understood by reference to the following detaileddescription of an illustrative embodiment when read in conjunction withthe accompanying drawing(s), wherein:

FIG. 1 is a diagram illustrative of the distance-based perspectiveeffect.

FIG. 2 is a flow chart of a method consistent with certain embodimentsof the present invention.

FIG. 3 is a further diagram illustrative of distance-based distortion.

FIG. 4 and FIG. 5 are flow charts of further methods consistent withcertain embodiments of the present invention.

FIG. 6 is a diagrammatic representation of a system consistent withcertain embodiments of the invention.

DETAILED DESCRIPTION

While this invention is susceptible of embodiment in many differentforms, there is shown in the drawings and will herein be described indetail one or more specific embodiments, with the understanding that thepresent disclosure is to be considered as exemplary of the principles ofthe invention and not intended to limit the invention to the specificembodiments shown and described. In the description below, likereference numerals are used to describe the same, similar orcorresponding parts in the several views of the drawings.

The distance-based perspective effect is illustrated in the diagram inFIG. 1. FIG. 1 shows a simplified camera 100. Light from an object 102at a first distance from the camera 100 passes through the lens 104 andfalls on sensor array 106 to form an image of height 108. Light from asecond object 110 passes through the lens 104 and falls on the sensorarray 106 to form an image of height 112. Although objects 102 and 110are of the same height, the image of the more distant object 102 is muchsmaller than the image of the closer object 110. Thus, objects (orportions of object) closer to the lens appear disproportionately largerin the image relative to more distant objects. The ratio of imageheights depends on the ratio of distances to the lens. Thus forthree-dimensional objects, such as faces, the amount of distortiondepends on the depth of the object relative to the distance of theobject to the camera lens. For many lenses in wide use, the distortionof the image of a face is therefore small when the face is severalmeters from the camera.

In some video applications, such as video-telephony, the user is able tomonitor the image of his or her face that is being transmitted. If theimage does not fill the image frame the user is likely to move closer tothe camera. This is especially true if the camera uses a wide anglelens. The result is that the image of the user's face, while fillingmore of the image frame, becomes distorted due to the distance-basedperspective effect. The resulting image is unflattering, showing anenlarged nose and pinched cheeks.

In the description below, an embodiment is described in which the objectis a face. However, the invention has application to the reduction ofdistance-based distortion in images of other objects and images ofmultiple objects.

FIG. 2 is a flow chart of a method consistent with certain embodimentsof the present invention. Referring to FIG. 2, following start block202, an image of a user is captured by a camera at block 204. At block206, an amount of camera zoom required is determined. The amount ofcamera zoom is determined such that when the user is sufficiently farfrom the camera that distance-based perspective distortion is acceptablysmall, the image of the user's face fills a significant portion of theimage frame. If the distortion is not acceptably small, as depicted bythe negative branch from decision block 208, the zoom is increased atblock 210 and a new image is displayed at block 212. The zoom may be adigital zoom, an optical zoom or a combination thereof. The increasedamount of zoom causes the image of the user's face to appear too largefor the image frame, thus prompting the user to move away from thecamera. The amount of zoom may be determined by standard distancemeasuring systems. These include, for example, active systems that useinfrared or ultrasonic waves to determine the distance to the user, orpassive systems based on the size of the image of user's face (takinginto account the current zoom), stereo images, or some other feature. Ifthe distortion is acceptably small (for example, if the user is at asufficient distance from the camera), as depicted by the positive branchfrom decision block 208, the image size is checked at block 214. If theimage size is not too large or too small, no further adjustment of thezoom is required. Flow continues to block 212 and the image isdisplayed. If the image size is too large or too small, as depicted bythe negative branch from decision block 214, the zoom is adjusted atblock 216 to provide the correct size and a new image is displayed atblock 212.

In this way, the image size is adjusted so the user is not prompted tomove closer to the camera. If the user is too close to the camera, sothat distance based distortion is unacceptably larger, the zoom isincreased so that image of the user's face appears too large for theimage frame, prompting the user to move farther from the camera, therebyreducing the distance-based distortion.

The region of the image corresponding to the user's face may beidentified so that the camera can zoom in on this region. Methods ofidentifying a user's face are well known in the art. The region may beidentified and tracked as the user moves, using techniques well known inthe art.

FIG. 3 is a diagram illustrating distance-based distortion. The objectis a stack of elliptical discs, shown in frontal view in view 300 withno distance-based distortion. The large disc 302 is farthest from thelens 104 of the camera 100 and small disc 304 is closest to the lens104. View 306 is a view of the image of the stack formed on the imagearray 106. The image 308 of the smallest (closest) disc is increased insize relative to the image 310 of the largest (farthest) disc. Acorresponding effect occurs with the image of a face close to thecamera.

If the distance from the lens 104 to a disc is known, the image of thedisc can be modified in size to compensate for the distance-baseddistortion. This same approach may be used with an image such as a face.

For a given object surface, the distance d from the camera lens alongthe optical axis is a function of the x and y coordinates in a planeperpendicular to the optical axis, so d=d(x,y). Light at position (x, y,d) meets the image sensor array at a position

${\left( {x^{\prime},y^{\prime}} \right) = \left( {\frac{\alpha \; x}{d\left( {x,y} \right)},\frac{\alpha \; y}{d\left( {x,y} \right)}} \right)},$

where α is a parameter dependent upon the optical characteristics (suchas the magnification) of the camera. The parameter α may be determinedby calibration. The image at position (x′,y′) on the sensor arraycorresponds to an object point at a distance

${d_{i}\left( {x^{\prime},y^{\prime}} \right)} = {d_{i}\left( {\frac{\alpha \; x}{d\left( {x,y} \right)},\frac{\alpha \; y}{d\left( {x,y} \right)}} \right)}$

from the camera. If the distance d_(i)(x′, y′) is known, a pixel valuep_(corrected) in a corrected image may be calculated from a pixel valuep_(image) in the captured image as

${{p_{corrected}\left( {{x^{\prime}\frac{d_{i}\left( {x^{\prime},y^{\prime}} \right)}{\beta}},{y^{\prime}\frac{d_{i}\left( {x^{\prime},y^{\prime}} \right)}{\beta}}} \right)} = {p_{image}\left( {x^{\prime},y^{\prime}} \right)}},$

where β is a scale factor. In one embodiment, the scale factor β is aconstant. The constant may be proportional to the mean value of d(x,y)or the root mean square value of d(x,y), for example. In anotherembodiment, the scale factor β=γ[d₀+d_(i)(x′,y′)] where d₀ is a constantdistance and γ is a scale factor. In this embodiment, a reprojected orvirtual image in obtained for which the face appears a distance d₀farther from the camera. In other words, a reprojected image of theobject is generated that is dependent upon the estimated distances andupon a virtual viewpoint that is more distant than the camera from theobject. A flow chart of an exemplary method for estimating the distanced(x,y) is shown in FIG. 4. Following start block 402, an image iscaptured at block 404. A face template or other model is matched to theimage at block 406. A variety of techniques for extracting features andfor matching templates or other models are known to those of ordinaryskill in the art. The face template may comprise, for example, simpleoval shapes for the periphery of the face and the eyes, and a simplenose template. The matching may be performed by use of alignmentmatching and verification, for example. The matching process results ina set of model parameters. At decision block 408, a decision is made asto whether the image includes a face. If a face is detected, as depictedby the positive path from decision block 408, the distance to one ormore matched features of the face is estimated at block 410. Thedistance may be estimated from the size of the matched templates, thedistance between the eyes, or other parameters of the templates.Alternatively, the distance could be determined by an active techniqueor by stereopsis. In one embodiment, the distances of various parts ofthe face are estimated from the template parameters at block 412. Forexample, a table may be accessed that contains typical distances for allregions of a face obtained from anthropological statistics. For example,a point 50% of the distance between the nose and the side of the faceand 20% of the distance from the center of the eye to the chin might be1 inch farther away than the tip of the nose. The distances may bedetermined for each position in the distorted image or for each positionin the undistorted face. Other distance estimating techniques are wellknown to those of ordinary skill in the art. At block 414, a correctedimage is formed using the captured image and the distance estimates foreach point in the image. The corrected image is displayed at block 416and the process terminates at block 418. The process may be repeated foreach consecutive image, or it may be performed when a change in faceposition is detected, or it may be repeated at periodic intervals.

FIG. 5 is a flow chart of a method consistent with certain embodimentsof the present invention. This embodiment uses pre-stored templates. Thetemplates are several views of the user photographed from differentangles (such as frontal and profile views). Following start block 502 inFIG. 5, an image is captured by the camera at block 504. At block 506 anumber of control points or features are identified in the capturedimage. For example, the position of a corner of an eye may be identifiedusing a method known in the art. The captured image is then warped ormorphed to match each template at block 508. This process matches thecontrol points in the captured image to corresponding control points inthe templates. At block 510, the best-matching template is selected.This image is displayed, transmitted and/or stored at block 512. Theprocess terminates at block 514. The goodness of a match between thewarped image and the template may be measured, for example, as acombination of an amount of deformation and an amount of residual colordifferences between the warped image and the template, where lessdeformation, less residual difference, or both correspond to a bettermatch. In an alternative embodiment, the templates are warped to matchthe image, and the best-matching template is selected. This template maybe used as the image to be displayed/stored, or the captured image canbe warped to match the selected template. Alternative embodiments willbe apparent to those of ordinary skill in the art.

In an alternative embodiment, the position of the image of a person'shead is detected using matching to templates or other means known in theart. Then a stored, template-independent, warping transformation isapplied to the image of the head. The transformation may beparameterized by the size and orientation of the detected image of thehead. For example, a geometric model of head, such as a spherical,cylindrical or ellipsoidal model, may be used. The size of the image ofthe head is used to estimate the distance of the head from the cameraand then the geometric model is used to estimate the variation ofdistance across the image of the head.

The methods described above with reference to FIGS. 2, 4 and 5 may beused in combination.

Examples of video cameras include a video camera of a videoconferencingsystem, a videophone, a video email product, a door camera and asecurity camera. The approach may also be used in still cameras, such ascameras in cellular telephone and other portable electronic devices.

FIG. 6 is a diagrammatic representation of a system consistent withcertain embodiments of the invention. Referring to FIG. 6, the systemincludes a camera 100, which may be a still camera or a video camera. Aprocessor 602 is coupled to the camera 100. The processor 602 may beincorporated in the body of the camera or may be external to the cameraand coupled to it by a wired or wireless link. In this embodiment, theprocessor also communicates with a display 604 and a communicationcircuit 606. In one embodiment, the processor operates to control thedigital or optical zoom of camera 100. Images captured from the cameramay be displayed on display unit 604 and/or transmitted viacommunication circuit 606 to a remote device.

In one embodiment of the invention, the camera 100 is a video camerathat captures images of the face of a user in a video conferencing orvideo telephony system. The display unit 604 displays images of the userand also images received from a remote camera. The processor 602controls the zoom of camera 100. If the user is so close to the camerathat a large amount of distance-based distortion is present in theimage, the zoom is controlled such that the image of the user on thedisplay is too large for the display frame, thereby encouraging the userto move farther from the display.

Additionally, the processor 602 may process images captured by thecamera to reduce distance-based distortion using the methods describedabove. This may be done by estimating distances from the camera to theuser's face or by matching templates to the image of the user's face. Onone embodiment, the distance is estimated using a distance measuringsystem 608, such an infrared or ultrasonic system. In anotherembodiment, the distance is estimated using a stereo vision system.

The system shown in FIG. 6 may be incorporated, for example, in a mobiledevice, such as a radio telephone, or in a fixed device, such as apersonal computer.

The present invention, as described in embodiments herein, may beimplemented using a programmed processor executing programminginstructions that are broadly described above in flow chart form thatcan be stored on any suitable electronic storage medium. However, thoseskilled in the art will appreciate that the processes described abovecan be implemented in any number of variations and in many suitableprogramming languages without departing from the present invention. Forexample, the order of certain operations carried out can often bevaried, additional operations can be added or operations can be deletedwithout departing from the invention. Error trapping can be added and/orenhanced and variations can be made in user interface and informationpresentation without departing from the present invention. Suchvariations are contemplated and considered equivalent.

Alternatively, the invention may be implemented in hardware.

The programmed processor or hardware may be integral to the camera ormay be a separate processor operating on images received from thecamera.

While the invention has been described in conjunction with specificembodiments, it is evident that many alternatives, modifications,permutations and variations will become apparent to those of ordinaryskill in the art in light of the foregoing description. Accordingly, itis intended that the present invention embrace all such alternatives,modifications and variations as fall within the scope of the appendedclaims.

1. A method for reducing distance-based distortion in an image of anobject, the image being captured by a camera, where the distance-baseddistortion is due to differences in distance from the camera todifferent parts of the object, the method comprising: for each templateof at least one template of the object, a camera determining a set ofwarping parameters that relate points in the image to correspondingpoints in the template of the object; selecting by the camera from theset of warping parameters for each of the at least one template, the setof warping parameters that results in the best match between the warpedimage and the corresponding template; and warping, by the camera, theimage using the selected set of warping parameters.
 2. A method inaccordance with claim 1, wherein the at least one template of the objectcomprise a plurality of views of the object.
 3. A method in accordancewith claim 1, wherein the object is a face and wherein the imagecomprises an image from a sequence of video images.
 4. A system forreducing distance-based distortion in an image of an object, the systemcomprising: a camera operable to capture an image of the object; anestimation element operable to estimate distance-based distortion in thecaptured image; a processing element operable to process the capturedimage to reduce distance-based distortion in the captured image.
 5. Asystem in accordance with claim 4, wherein the estimation element isoperable to estimate the distance to at least one region of the object.6. A system in accordance with claim 4, wherein the estimation elementis operable to estimate the size, in the image, of a specified featureof the object.
 7. A system in accordance with claim 4, wherein theestimation element comprises an active distance measuring means.
 8. Asystem in accordance with claim 4, wherein the estimation element isoperable to estimate the amount of warping needed to match points in thecaptured image to corresponding points in a previously stored image. 9.A system in accordance with claim 4, wherein the processing element isoperable to warp the captured image such that points in the capturedimage match points in a previously stored image.
 10. A system inaccordance with claim 4, wherein the processing element is operable toreproject the captured image to simulate a more distant camera viewpoint.
 11. A system for reducing distance-based distortion in an imageof an object, the system comprising: a camera operable to capture animage of the object; a means for estimating distance-based distortion inthe captured image; a means for processing the captured image to reducedistance-based distortion in the captured image.
 12. A system inaccordance with claim 11, wherein the means for estimatingdistance-based distortion in the captured image comprises a means forestimating the distance to at least one region of the object.
 13. Asystem in accordance with claim 11, wherein the means for estimating thedistance to at least one region of the object comprises a means forestimating the size, in the image, of a specified feature of the object.14. A system in accordance with claim 11, wherein the means forestimating the distance to at least one region of the object comprisesan active distance measuring means.
 15. A system in accordance withclaim 11, wherein the means for estimating distance-based distortion inthe captured image comprises a means for estimating the amount ofwarping needed to match points in the captured image to correspondingpoints in a previously stored image.
 16. A system in accordance withclaim 11, wherein the means for processing the captured image to reducedistance-based distortion in the captured image comprises a means forwarping the captured image such that points in the captured image matchpoints in a previously stored image.
 17. A system in accordance withclaim 11, wherein the means for processing the captured image to reducedistance-based distortion in the captured image comprises a means forreprojecting the captured image to simulate a more distant camera viewpoint.